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AGENT-BASED  COOPERATIVE  CONTROL 


1.  Introduction 

This  research  has  sought  to  formulate  a  general  theoretical  framework  for  the  cooperative  control  of 
multiple  dynamic  systems  or  units .  There  are  two  classes  of  cooperative  control  problems:  those 
concerning  manipulators  and  those  concerning  mobile  robots;  this  proposed  work  is  with  regard  to  the 
latter.  The  most  straightforward  examples  of  such  systems  are  groups  or  teams  of  autonomous  vehicles 
cooperating  to  achieve  a  common  goal.  The  “cooperative”  aspect  of  the  problem  implies  that  no  benefit  is 
derived  by  the  team  if  only  a  single  vehicle  performs  a  task,  and  that  a  high  degree  of  tightly  coupled , 
coordinated  action  is  required.  The  class  of  problems  considered  generically  in  this  proposal  therefore 
requires  the  simultaneous  action  of  multiple  vehicles  in  a  coordinated  fashion. 

Interest  in  cooperative  control  of  mobile  robots  has  been  heightened  in  recent  years  mainly  due  to  future 
planned  scenarios  involving  multiple  Unmanned  Aerial  Vehicles  (UAVs)  for  the  United  States  Air  Force. 
Despite  the  heightened  research  activity  in  this  area,  significant  issues  with  regard  to  both  theory  and 
implementation  still  exist.  Theoretically,  much  work  has  been  done  regarding  different  aspects  of  the 
problem,  but  few  if  any  attempts  have  been  made  to  formulate  a  theoretical  framework  that  would  unify 
these  somewhat  separate  efforts.  The  question  then  arises,  “Why  is  establishing  a  theoretical  framework 
worth  it?  What  does  it  buy  you?” 

The  overarching  philosophy  behind  this  work  is  the  following:  Significant  implementation 
issues  exist  with  regard  to  the  cooperative  control  of  multiple  autonomous  vehicles .  Establishing 
a  theoretical  framework  that  is  capable  of  universally  and  analytically  treating  low,  mid  and 
high  level  planning  and  control  is  a  necessary  step  toward  addressing  such  implementation 
issues.  The  developed  framework  will  allow  questions  regarding,  but  not  limited  to, 
communicated  information  requirements,  robustness  to  communication  delays  and 
asynchronous  communication,  input  saturation  constraints,  and  non-holonomic  behavior,  to  be 
posed  and  addressed  mathematically. 

The  proposed  work  is  rooted  firmly  in  dynamical  systems  and  model-based  control  theory  in  an  effort  to 
addresses  issues  that  have  not  previously  received  much  attention  -  namely  stability  and  robustness 
within  a  well-founded  continuous-time  analytical  control  structure.  This  is  somewhat  of  a  departure  from 
the  vast  amount  of  work  currently  being  done  in  cooperative  control  from  a  computational  and  network 
communications  protocol  point-of-view.  Continuation  of  this  work  will  help  determine  the  minimum 
requirements  for  these  difficult  technological  and  system  architecture  issues  by  providing  an  efficient, 
well-founded  structure  to  control  such  systems. 

As  will  be  shown  for  a  number  of  applications,  it  will  be  useful  and  desirable  to  achieve  the  necessary 
coordination  or  cooperation  without  the  use  of  a  central  controller/planner.  In  the  language  of  control,  this 
amounts  to  not  casting  the  problem  as  a  multivariable  control  problem  where  the  control  law  is  computed 
with  full  state  information  of  all  the  vehicles  under  consideration.  Therefore,  the  challenge  is  to  define  the 
interaction  among  the  vehicles,  or  units,  in  a  non-centralized  way  that  divides  the  task  between  either 
identical  (homogeneous),  or  specialized  (heterogeneous)  units,  such  that  each  unit  can  exercise  a  local 
control  law  based  on  local  information  (the  unit’s  own  state,  and  information  exchanged  between  units) 
that  contributes  toward  achieving  a  global  goal  (minimizing  a  team-wide  cost).  In  this  manner,  the  units 
will  serve  as  “agents”  of  a  team  policy  that  is  cooperative  and  distributed.  The  “agent-based”  approach 
proposed  therefore  seeks  to  formulate  the  cooperative  control  problem  such  that  local  control  laws  can 
be  exercised  by  the  units,  while  still  enforcing  an  analytically  guaranteed  team  policy  with  all  the 
assurances  of  stability,  robustness  and  performance  normally  sought  in  the  field  of  control  theory. 
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Fundamentally,  this  research  seeks  to  help  answer  the  following  general  open-ended  questions  regarding 
the  theoretical  challenges  presented  by  cooperative  control: 

•  Can  a  cohesive  analytical  framework  be  developed  for  the  control  of  multiple  dynamic  systems 
that  provides  analytical  assurances  of  stability,  robustness,  convergence,  and  other  quantifiable 
metrics,  regarding  both  the  team  and  the  individual  units? 

•  What  is  an  appropriate  set  of  principles  upon  which  to  formulate  this  framework?  Can  an  ad-hoc 
set  of  rules  be  avoided  in  favor  of  fundamental  principles? 

•  If  so,  will  a  framework  grounded  in  fundamental  principles  reveal  a  minimalist  structure 
regarding  the  required  information  exchange  and  communication  among  the  dynamic  units? 

•  If  so,  can  the  overall  problem  be  defined  in  a  manner  that  is  general  enough  so  that  the  resulting 
framework  applies  to  a  broad  class  of  problems? 

To  help  answer  these  and  other  related  questions,  a  framework  will  be  developed  such  that  well 
established  analysis  tools  can  be  utilized  to  show  robust  and  stable  control  on  both  the  local  and  global 
level.  In  brief,  the  core  idea  of  this  proposed  framework  will  be  to  interpret  the  following  three 
fundamental  principles  in  a  unifying  manner  within  the  requirements  of  the  class  of  cooperative  control 
problems  under  consideration:  1.  control  Lyapunov  functions,  2.  potential  field  theory,  3.  the  optimal 
return  or  value  function  from  dynamic  programming.  This  core  idea  follows  from  an  assumption  that 
stable,  robust  and  effective  cooperative  control  must  follow  from,  and  include,  considerations  of  stability 
and  robustness  of  each  individual  unit.  So  the  reader  may  now  appropriately  ask,  “So  what’s  new  here, 
and  how  is  this  not  just  three  ideas  in  a  blender?  What  will  be  the  real  contribution?”  This  bottom-up 
approach  is  not  hierarchical ,  but  rather  it  is  unified  in  nature .  It  will  unveil  the  amount  of  information 
exchange  required ,  remove  the  ambiguity  surrounding  closing  control  loops  at  various  levels,  and 
provide  analytical  assurances  of  stability,  robustness  and  performance . 


THE  CORE  IDEA:  Unification  of  three  principles  to  form  a  theoretical  framework  for  cooperative  control: 


Control  Lyapunov  Function 

Addresses:  issues  of  stability  and 
robustness  through  model-based 
nonlinear  control  theory. 

/— N 
W 

Potential  Field 

Addresses:  issues  of  desired 
interaction  such  as 
coordination,  formation, 
and  collision  avoidance. 

/— N 
W 

Optimal  Return  Function 

Addresses:  issues  regarding 
assignment  -  what  task  among 
several  the  team  performs,  and 
which  nearby  vehicles  participate. 

The  unification  of  these  three  principles  will  guide  a  well-founded  and  fundamental  inclusion  of  the 
following  considerations: 

•  Individual  vehicle  dynamics  (including  non-holonomic  dynamics  and  constraints) 

•  Minimal  required  inter- vehicle  communication  and  information  exchange 

•  Low-level  control  for  path  following  and  trajectory  tracking 

•  Collision  avoidance  with  environmental  obstacles  and  other  vehicles 

•  Cooperative  precision  configuration/formation  and  pursuit/evasion/search  team  control 

•  Vehicle  assignment  (forming  teams  from  available  vehicles  for  lowest  cost  completion) 

The  core  idea  will  allow  all  of  the  above  considerations  to  be  cast  within  a  single  unified  structure  so  that 
a  rigorous  analysis  of  stability,  performance  and  robustness  of  the  entire  system  of  cooperating  units  can 
be  made.  Put  another  way:  the  core  idea  will  allow  all  of  these  operational  considerations  to  be  viewed  in 
the  same  fundamental  manner.  Low-level  control,  desired  real-time  trajectory  generation,  vehicle 
cooperation  and  interaction,  and  task  assignment  will  all  be  able  to  be  considered  simultaneously. 
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2.  Literature  Review 


In  reviewing  the  scientific  literature,  there  are  four  key  areas  that  have  made  contributions  relevant  to  the 
cooperative  control  problem  class.  These  areas  are:  artificial  potential  fields  for  obstacle  avoidance, 
motion  planning  for  nonholonomic  vehicles,  swarm  formation  control,  and  the  optimal  assignment  of 
vehicles  to  tasks.  Although  the  work  in  these  areas  has  been  quite  significant  and  extensive,  the 
integration  of  these  contributions  within  the  context  of  cooperative  control  is  not  mature.  Establishing  a 
framework  for  cooperative  control  will  bind  together  these  four  disparate  areas  of  research.  Although 
some  important  exceptions  can  be  found,  artificial  potentials  have  typically  not  been  applied  to 
cooperative  problems,  swarm  formation  control  is  typically  unstructured  and  not  highly  coordinated, 
motion  planning  for  nonholonomic  vehicles  typically  involves  only  a  single  vehicle,  and  optimal 
assignment  is  typically  performed  open-loop  as  a  static  optimization  problem  and  not  cast  in  continuous¬ 
time.  The  proposed  framework  will  tie  together  these  areas  under  one  umbrella  to  enable  the  design  and 
analysis  of  cooperative  control  systems  utilizing  either  homogenous  or  heterogeneous  vehicles. 

Khatib  [40,41,42]  pioneered  the  use  of  artificial  potential  fields  for  obstacle  avoidance  of  manipulators  and 
mobile  robots.  Goal  points  were  set  as  attractors  and  obstacles  were  set  as  repulsive,  and  the  desired  path 
was  the  resulting  gradient  of  this  combined  field.  The  gradient  was  tracked  using  feedback  linearization. 

A  recognized  drawback  of  this  initial  approach  was  that  exact  knowledge  of  the  dynamics  were  required 
to  ensure  stability  (in  other  words  the  method  was  not  robust).  Koditschek  [45,46]  modified  this  approach  by 
applying  energetic  arguments  to  provide  robustness  to  a  class  of  dissipative  mechanical  systems.  Newman 
and  Hogan  [70]  built  upon  the  work  of  Khatib  for  obstacle  avoidance  for  robot  manipulators.  They 
explicitly  built  joint  torque  constraints  into  the  potential  field  by  defining  a  time  varying  potential  based 
on  energetic  arguments.  Rimon  and  Koditschek  po]  proposed  a  method  of  constructing  potential  field- 
based  navigation  functions  for  robot  navigation  around  obstacles  that  result  in  a  bounded  torque  feedback 
controller.  Guldner  and  Utkin  [32,33]  proposed  a  potential  based  method  for  obstacle  avoidance  by  a 
holonomic  robot.  The  key  feature  of  the  method  was  that  the  control  law  is  not  based  directly  on  the 
gradient  of  the  potential  field,  but  rather  the  gradient  lines  were  interpreted  as  the  desired  trajectories.  A 
sliding  mode  approach  is  then  adopted  to  converge  to  a  sliding  surface  that  takes  the  robot  to  the  gradient 
line  and  provides  exact  tracking  once  on  the  gradient  line.  This  modification  to  Khatib ’s  early  approach  of 
deriving  the  control  law  directly  from  the  potential  field  provides  robustness  to  modeling  errors.  In  this 
manner,  the  work  of  Guldner  and  Utkin  forms  a  link  between  a  control  Lyapunov  function  and  a  potential 
field.  Furthermore,  it  establishes  a  method  that  decouples  the  design  of  the  potential  field  from  the  design 
of  a  controller  to  track  gradient  lines.  The  method  however  is  presented  for  a  single  robot  and  with  non¬ 
time  varying  potential  fields;  how  to  extend  this  work  to  include  interaction  between  two  or  more  robots 
with  their  own  potentials  plus  an  environmental  potential  is  unclear. 

The  early  work  of  Dub  ins  [22]  established  interest  in  the  static  optimization  problem  of  trajectory  or  path 
planning  for  a  single  non-holonomic  vehicle  with  minimum  turning  radius  constraints.  Murray  [67]  and 
Tilbury  [99]  (sinusoids),  Leonard  [56],  Bloch  [13,14,15],  andNair  [69]  (controlled  Lagrangians),  Nair  [68]  and 
Astrom  [2]  (energy  approaches),  and  others  [52,104]  have  also  tackled  various  aspects  of  the  constrained  input 
problem.  With  regard  to  real-time  path  planning,  Primbs  et  al  [81,82,26]  offer  some  very  keen  observations 
regarding  the  connections  between  the  value  function  of  the  Hamilton-Jacobi-Bellman  equation,  control 
Lyapunov  functions  (CLF),  and  minimum  norm  controllers  through  Sontag’s  formula.  This  work  begins 
to  make  some  of  the  desired  connections  between  CLF’s,  potential  fields  and  the  optimal  return  function 
sought  by  this  proposed  work.  Essentially,  the  paper  points  out  that  the  gradient  of  a  CLF  is  simply  some 
scaling  of  the  gradient  of  the  value  function  (or  optimal  return  function)  from  the  control  law  solution  of 
the  HJB  equation,  V*  =  A,(x)Lx .  It  can  also  be  shown  that  a  given  CLF,  V,  corresponds  to  the  value 

function,  V * ,  for  some  set  of  cost  functions  (the  so-called  “inverse  optimal”  property).  The  dynamic 
window  approach  to  obstacle  avoidance  in  Ogren  and  Leonard  [72]  uses  the  connections  pointed  out  by 
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Primbs  to  beautifully  merge  aspects  of  model  predictive  control  and  control  Lyapunov  functions  to 
provide  a  theoretical  assurance  of  convergence  for  the  navigation  of  a  single  mobile  robot.  Essentially,  the 
result  is  a  potential  field  method,  through  the  construction  of  a  navigation  function,  with  assurances  of 
convergence.  The  method  neatly  addresses  the  concern  that  arises  when  simply  utilizing  the  gradient  lines 
of  the  potential  field  as  desired  trajectories  thereby  making  them  state  dependent  and  in  turn  no  longer 
exogenous  inputs  and  hence  spoiling  the  stability  and  robustness  of  using  a  Lyapunov  method  to  track 
them.  Instead,  the  navigation  function  (potential  function)  is  incorporated  into  the  CLF  directly  along  with 
an  additional  more  traditional  tracking  error  term:  V (r,  f )  =  \rTr  +  NF (r) ,  V  =  r Tu  +  rTVNF (r)  <  -s||f  || 

where  the  plant  is  given  by  r  =  u  .  Although  well  prescribed,  the  construction  of  the  navigation  function  is 
performed  via  solving  the  shortest-path  problem  cell-wise  on  an  obstacle  grid  map,  and  hence  must  be 
computed  ahead  of  time  (i.e.  not  in  real-time).  The  method  also  leaves  questions  regarding  establishing  a 
connection  between  the  CLF  and  the  optimal  return  function.  Within  the  realm  of  cooperative  control  of 
mobile  robots,  work  has  been  done  on  formation  and  swarm  control  utilizing  potential  field  ideas  [t35,tso,t57], 
employing  stability  arguments  [58,27,65,97],  graph  theory  [21,74],  invariant  manifold  techniques  [78],  decentralized 
approaches  [54,62,55],  leader-follower  approaches  [48],  platooning  [31,103,36,84,95],  and  hybrid  and  multi-modal 
control  approaches  [25,106,50,49]. 

The  assignment  of  multiple  available  vehicles  to  various  tasks  has  been  approached  by  many  including 
Passino  [so],  Beard  [5,6],  Liu  [60],  Chandler  [17],  Schumacher  [93],  Pachter  [76],  and  others  [66, 83,63,71],  but  typically 
the  problem  is  approached  from  a  non-dynamical  systems  point  of  view.  Cooperative  path  planning  via 
mixed  integer  linear  programming  is  a  popular  non-dynamical  systems  approach  [89,94].  Various  other 
approaches  begin  to  make  some  connections  between  an  optimal  return  type  function  for  decisions  and  a 
team  metric  akin  to  a  CLF,  for  assignment  type  problems:  satisficing  approaches  [85,77,24,20,38],  Lyapunov 
functionals  [35],  Lyapunov  certificates  [ioi],  probability  of  loss  [9],  behavior-based  approaches  [92,3],  consensus 
approaches  [75,88],  biologically  inspired  strategies  [29,59],  and  auction  methods  [30].  The  unification  of  vehicle 
dynamics  and  real-time  cooperative  path  planning  and  assignment  remains  a  difficult  challenge. 

Although  significant  in  their  achievements ,  there  has  been  no  effort  among  prior  works  to  unify  all  of 
the  above  concerns  (namely:  model-based  control  theory ',  cooperative  interaction,  real-time  path 
planning,  and  assignment)  into  a  single  cohesive  theoretical  framework. 

3.  Proposed  Control  Framework 

As  stated  in  the  introduction,  the  objective  of  the  proposed  research  is  to  formulate  a  general  theoretical 
framework  to  address  the  issues  unique  to  cooperative  control.  The  approach  to  be  taken  is  to  unify  the 
fundamental  principles  of  control  Lyapunov  functions,  potential  field  theory,  and  the  so-called  optimal 
return  function.  The  flowchart  in  Figure  1  shows  how  these  three  principles  will  be  woven  together  (left- 
hand  column)  to  achieve  an  analytically  rigorous  formulation  that  addresses  the  required  functionality 
presented  by  cooperative  control  problems  (shown  in  the  right  hand  column). 

The  framework  consists  of  four  principle  elements:  the  optimal  return  function  (ORF),  the  cooperative 
control  Lyapunov  function  (CCLF),  the  Potential  Function  (PF),  and  the  agent  control  function  (ACF). 

The  CCLF  ensures  that  the  team  asymptotically  reduces  the  team  cost.  The  time  derivative  of  the  CCLF  is 
split  among  the  agents  with  an  associated  ACF  being  assigned  to  each  agent,  whereby  this  step  defines  an 
agent  and  its  role.  The  ACF’s  incorporate  PF’s  to  dictate  local  agent-to-agent  interaction  (e.g.  collision 
avoidance),  and  to  accommodate  dynamic  and  input  constraints  (e.g.  nonholonomic  and  control 
saturation).  Effectively  the  PF  specifies  a  portion  of  each  agent’s  desired  trajectory,  while  the  other 
portion  comes  from  the  CCLF.  If  the  time  derivative  of  each  ACF  can  be  selected  negative  definite,  then 
the  CCLF  will  be  negative  definite  in  a  sufficient  sense  implying  the  team  and  each  agent  will  be  assured 
to  be  asymptotically  stable.  If  the  time  derivative  of  every  ACF  is  not  selected  negative  definite,  but 
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rather  in  a  way  such  that  all  ACF’s  combine  to  form  a  negative  definite  time  derivative  of  the  CCLF,  then 
the  team  is  ensured  asymptotically  stable  but  each  agent  is  only  ensured  to  be  bounded-input  bounded- 
output  stable.  This  second  case  will  give  the  team  the  ability  to  make  individual  sacrifices  (non-optimal 
for  an  individual  agent)  for  the  sake  of  benefiting  the  team.  Finally,  the  ORF  represents  the  optimal  “cost- 
to-go”  and  will  be  related  analytically  to  the  CCLF.  If  the  CCLF  is  viewed  as  a  generalized  distance  to  the 
goal,  and  the  time  derivative  of  the  CCLF  (the  generalized  velocity  toward  the  goal)  is  designed  to  be  a 
particular  analytical  function,  then  the  generalized  time  to  reach  the  goal  can  be  solved  in  closed  form. 

The  ORF  is  therefore  the  optimal  return  function  for  a  particular  team  task.  Various  team  tasks  and  their 
associated  ORF’s  can  then  be  evaluated  in  order  to  select  the  optimal  task  for  the  team  from  among  a 
discrete  set  of  possible  team  tasks  (optimal  in  the  sense  of  the  ORF  plus  the  cost  to  switch  tasks).  Various 
combinations  of  different  vehicles  available  to  be  assigned  to  a  task  (the  assignment  problem)  can  then  be 
selected  by  evaluating  ORF’s  for  each  combination. 


Unified  Approach  to  Agent-Based  Cooperative  Control 


Principle  Elements  Functionality 


Figure  1:  Flowchart  of  the  main  principles,  connections  between  them,  and  resulting  functionality 
in  the  proposed  agent-based  cooperative  control  framework.  The  framework  is  unified  in  the  sense 
that  the  global  convergence,  stability,  and  robustness  is  ensured  by  virtue  of  the  stmcture  of  the 
framework. 

3.1  Scientific  Relevance 

The  proposed  development  of  a  framework  for  cooperative  control  seeks  to  add  an  analytical  structure  to 
a  relatively  young  research  topic  with  applications  extremely  important  to  the  nation.  By  seeking  to  base 
this  framework  in  fundamental  principles,  ad-hoc  and  heuristic  approaches  will  be  able  to  be  replaced 
with  methods  that  will  provide  solid  analysis  and  well  understood  control  design.  Further,  by  combining 
the  as  yet  disparate  areas  of  artificial  potential  fields,  swarm  formation  control,  motion  planning  for 
nonholonomic  vehicles,  and  optimal  assignment  of  vehicles  to  tasks,  with  a  unique  unifying 
interpretation,  this  work  will  help  lay  the  groundwork  for  an  emerging  branch  of  controls  -  cooperative 
control. 
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3.2  Relevance  to  the  Air  Force 


In  addition  to  formulating  a  generic  approach  that  applies  to  a  large  class  of  problems  under  the  umbrella 
of  cooperative  control,  the  proposed  research 
program  will  also  address  a  number  of  specific 
applications.  A  currently  existing  platform  awaiting 
future  conceived  cooperative  Unmanned  Aerial 
Vehicle  (UAV)  missions,  is  the  UCAVs  pictured  in 
Figure  2.  Below  is  a  list  of  military  (see  attached 
letter  of  support)  and  non-military  cooperative 
control  applications  which  are  planned  to  be 
specifically  addressed  by  the  developed  framework. 

Selected  applications  in  bold  below  with  be 
formulated  and  simulated  as  a  simplified  planar  problem,  experimentally  verified  with  non-holonomic 
wheeled  platforms  in  the  lab,  and  then  extended  to  the  6-DOF,  3-D  version  in  collaboration  with  the  Air 
Force  and  its  extensive  simulation  software  (non-classified  and  available  for  university  use). 


Figure  2:  Boeing  X-45A  Uninhabited  Combat  Aerial 
Vehicles  (UCAV).  A  pair  on  the  left,  and  flying 
alongside  a  manned  F- 16  on  the  right. 


Military  Applications 

In  the  CGMTE  (Cooperative  Ground  Moving  Target  Engagement)  scenario,  two  or  more  UAV’s 
must  circle  a  ground  target  such  that  their  sensor  footprints  overlap  in  a  particular  configuration  to 
provide  adequate  fused  accuracy  to  localize  the  target.  The  cooperative  control  laws  enacted  must  fly  the 
UAV’s  to  maintain  this  sensor  fusion  requirement,  while  accommodating  non-holonomic  dynamics, 
turning  rate  constraints,  and  avoiding  collisions.  In  the  Cooperative  Visual  Target  Tracking  scenario,  it 
is  envisioned  that  multiple  UAV’s  will  track  targets  visually.  Imagine  a  single  UAV  visually  locating  a 
ground  target  moving  among  non-targets.  In  order  to  track  the  target  with  additional  UAV’s,  each 
additional  UAV  must  first  acquire  a  visual  from  a  similar  perspective  and  range.  This  hand-off  will 
require  UAV’s  to  cooperatively  fly  very  near  each  other.  The  similar  problems  of  in-flight  refueling  with 
autonomous  vehicles  (a  heterogeneous  problem)  and  cooperative  laser  designation  will  also  be 
pursued.  The  so-called  SEAD  (Suppression  of  Enemy  Air  Defenses)  Missions  destroy,  deceive  or  jam 
an  enemy’s  Integrated  Air  Defense  Systems  (IADS).  IADS  typically  consist  of  a  number  of  ground-based 
tracking  radar  stations  (that  compare  data  in  real-time  to  enhance  their  detection  abilities)  and  a  number  of 
surface-to-air  missile  (SAM)  launch  sites.  To  destroy  an  IADS,  it  is  envisioned  that  one  or  more  UAV’s 
will  jam  appropriate  radar  stations  while  another  masked  aircraft  bombs  the  SAMs.  To  deceive  an  IADS, 
multiple  UAV’s  must  cooperatively  present  a  phantom  aircraft  radar  image  by  sending  delayed  radar 
pulses  to  specific  radar  stations  (the  UAV’s  are  assumed  radar  invisible).  If  the  phantom  image  is  not 
adequately  correlated,  the  IADS  will  detect  that  it  is  a  deception.  To  cooperatively  jam  an  IADS, 
multiple  UAV’s  must  shrink  the  detection  range  of  radar  stations  cooperatively  in  order  to  route  a  non- 
radar-invisible  aircraft  through  the  IADS  region.  Finally  the  Convoy  Escort  problem  is  one  where  a 
convoy  of  ground  vehicles  must  be  protected  by  monitoring  for  enemies  on  its  flanks  and  ahead  of  the 
convoy.  Given  that  UAV’s  fly  much  faster  than  the  ground  vehicles  being  escorted,  they  must 
cooperatively  switch  off  monitoring  one  of  the  two  sides,  or  the  front,  in  a  manner  such  that  there  are 
minimal  gaps  in  the  monitored  regions. 

Non-Military  Applications 

Listed  below  are  also  a  number  of  closely  related  but  non-military  autonomous  cooperative  scenarios 
imagined  for  mobile  robots.  In  coordinated  search  and  rescue  it  envisioned  that  multiple  autonomous 
ground  vehicles  or  aerial  vehicles  will  cooperatively  search  for  victims  of  such  situations  as  fallen 
buildings  or  sea  search  and  rescue.  This  application  presents  the  opportunity  to  combine  potential  field 
ideas  and  probability  density  functions  to  achieve  effective  searching.  Cooperative  forest  and  brush  fire 
suppression  presents  similar  theoretical  development  opportunities.  Cooperative  monitoring  applications 
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such  as  border  patrol  and  atmospheric  profiling  mirror  the  motivations  of  the  military  convoy  escort 
problem  above.  Problems  associated  with  intelligent  highway  systems  also  represent  the  type  of  tightly 
coupled  coordination  sought  to  be  addressed  by  the  proposed  framework. 

4.  Technical  Approach  -  Initial  Results  and  Discussion 

This  section  provides  a  preliminary  investigation  done  to  show  the  viability  of  the  proposed  framework. 
Specifically,  this  preliminary  work  demonstrates  connections  between  the  notion  of  potential  fields  and 
control  Lyapunov  functions.  It  also  shows  the  relationship  between  the  CCLF  and  the  formation  of  the 
ACF’s. 

Figure  3  below  shows  a  bit  more  detail  regarding  the  connections  between  the  OR F  serving  as  an  estimate 
for  the  cost  of  the  team  completing  a  task  or  satisfying  and  holding  a  certain  configuration,  the  CCLF 
serving  as  a  means  of  always  reducing  the  total  team  cost,  the  PF  serving  as  a  means  of  incorporating 
interaction  and  vehicle  constraints,  and  the  ACF’s  serving  to  collectively  decrement  the  CCLF  under  the 
constraints  presented  by  the  PF. 


ACF 

Agent  Control  Function 

VACFi  =  PF  &  CCLF  contrib.  to  agent  i. 


V  =  V  x  = 

v  ACFi  v  x,  V 


L. 


Some  chosen  form 
(resulting  from  CCLF) 


Agent  1 


Agent  2 


Agent  n 


System  dynamics  of  agent  /. 

Gradient  of  “potential”  felt  by  agent  i. 

_ _ _ x 

A  Subdivide  VCCLF  into  VACF  ’s  to: 

1.  Incorporate  interaction  PF’s. 

2.  Minimize  communication  in  each  agent 
seeking  its  prescribed  VACLi . 

3.  Have  only  agent  V  s  control  u  in  VACFi  • 

CCLF 

Cooperative  Control  Lyapunov  Function 

LF  =  Team’s  current  “distance”  from  “goaf 

LF  =  Team’s  rate  of  closing  the  “distance”  to  goal 
=  Optimal  Return  Function  =  Cost  to  get  to  goal 


Regularize  Optimal  Return  Functions 


Design  negative  definite  function 
so  that  the  Optimal  Return 

Function  v*  can  be  computed 
analytically  at  time  t. 


V*  can  be  the  area  under  the  curve 
(or  other  integral  form).  Or,  it  can 
be  the  time  for  the  CCLF  to  reach 
a  certain  low  value  (or  other 
solution  to  a  differential  equation). 


ORF 

Optimal  Return  Function 


One  v*  for  each  task  and  for  each  discrete  team  grouping  per  task.  The  optimal  v*  is 
selected  using  Dynamic  Programming.  The  cost  of  switching  between  tasks  or  switching 
which  group  of  vehicles  does  the  task  is  included  in  the  consideration. 


Figure  3:  Connection  diagram  showing  the  unification  of  the  principles  of  Lyapunov  control 
functions,  potential  fields  and  the  optimal  return  function. 
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4.1  Relating  the  Potential  Field  (PF),  the  Cooperative  Control  Lyapunov  Function  (CCLF)  and  the 
Agent  Control  Functions  (ACF’s) 


Agent  Control 

/ — \ 

Potential  Field  (PF) 

/ — \ 

Cooperative  Control 

Function  (ACF) 

\ — / 

\ — / 

Lyapunov  Function  (CCLF) 

This  example  will  show  how  three  components  of  the  proposed  framework  begin  to  interrelate  and  fit 
together,  namely  the  potential  field  (PF),  the  Cooperative  Control  Lyapunov  Function  (CCLF)  and  the 
Agent  Control  Function  (ACF).  Being  preliminary  work  only,  this  example  will  of  course  not  definitively 
resolve  general  issues  regarding  these  three  components  of  the  approach,  but  rather  it  serves  as  a  jumping- 
off  point  to  begin  to  expose  the  critical  issues  that  must  be  formally  and  theoretically  addressed  in 
pursuing  the  proposed  framework. 

In  order  to  illustrate  how  the  PF  can  be  used  to  separate  the  cooperative  control  task  into  “agents” 
representing  the  multiple  vehicles,  and  how  each  agent  can  be  symmetric  (i.e.  look  identical  to  other 
agents)  and  exercise  a  local  control  law  (from  the  ACF)  that  contributes  constructively  along  with  other 
ACF’s  to  minimizing  the  global  cost  (from  the  CCLF),  consider  the  following  example.  Consider  the 
simplified  problem  of  two  cooperating  non-holonomic  vehicles  that  must  establish  a  particular  orbit 
around  a  target  with  a  particular  angular  separation  between  the  two  vehicles.  This  problem  is  considered 
as  a  simplified  first  step  toward  solving  the  Cooperative  Ground  Moving  Target  Engagement  (CGMTE) 
problem.  The  scenario  considered  is  shown  in  Figure  4  below. 


Figure  4:  Cooperative  Ground  Moving  Target  Engagement  (CGMTE)  problem  showing  the 
cooperative  requirements  of  two  UAV’s.  The  UAV’s  must  be  within  the  Doppler  visibility  cone, 
with  their  sensor  footprints  over  the  target,  and  with  the  error  detection  ellipses  of  each  sensor 
contributing  to  a  more  precise  and  accurate  localization  of  the  target  than  achievable  by  either 
UAV  individually. 

Let  us  consider  a  two  dimensional  representation  of  the  problem  where  the  planar  dynamics  of  each 
vehicle  are  given  by  the  so-called  Dubin’s  car  model  [22] : 

x  =  vcos(0) ,  y  =  vsin(0) ,  0  =  u  (la) 

where  x  andy  are  the  Cartesian  coordinates  of  the  vehicle,  6  is  the  heading  angle,  v  is  the  velocity  of  the 
vehicle  (assumed  constant),  and  u  is  the  control  variable  representing  the  heading  angle  rate  of  change. 
This  can  be  represented  more  compactly  as  x  =  f  (x)  +  b(x)w  : 
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\u 


(lb) 


X 

Xi 

vcos(x3) 

'o' 

y 

= 

x2 

= 

vsin(x3) 

+ 

0 

0 

_x3J 

0 

1 

Two  vehicles  are  then  represented  by  x1  =  f{(xx)  +  b1(x1)t/1  and  x2  =f2(x2)  +  b2(x2)w2 .  To  establish  the 
required  positions  of  the  two  vehicles  cooperatively,  three  error  metrics  are  required.  The  first  two  metrics 
pertain  to  each  vehicle  being  at  a  prescribed  radius  rd  from  the  ground  target.  Additionally,  in  order  to 
place  the  vehicles  at  90  degrees  of  each  other  for  the  proper  sensor  fusion,  the  distances  between  the 
vehicles  must  be  maintained  as  rd4l  .  To  satisfy  the  first  two  requirements,  we  wish  to  drive  the 
following  error  to  zero  for  each  vehicle  ( ex  and  e2 ): 

<?i  =  (A2  - r2d)  and  e2  =  (r22  - r2d)  (2) 

For  each  individual  vehicle,  note  the  following  error  dynamics  in  terms  of  the  system  dynamics: 

e  =  2rr-2rdrd  (3) 

e  =  2 f 2  +  2 rr  -  2 r2  -  2 rdrd  (4) 

Where  r  =  yjx2  +  y2  ,  f  =  (x2  +  y2)  2  (xx  +  yy) ,  and  the  input  u  appears  in, 

2 rr  =  -2r  2  (xx  +  yy)2  +  2(x2  +  y 2 )  +  2(xy  -  xy)u  (5) 


To  satisfy  the  coupling  requirement  for  the  vehicles  being  a  certain  distance  apart,  we  wish  to  drive  the 
relative  separation  distance  error  to  zero: 

er  =  O2  -  f  +  O2  -  Ti  )2  -  2rd  (6) 

Note  the  following  relative  separation  error  dynamics: 

er  =  2(xl  -  x2 )(x1  -x2)  +  2{yx  -  y2)(yx  -y2)  (!) 

er=2(xx  -x2)2  +2(Xj  -x2)(-ylul  +y2u2)  +  2(yx  -y2f  +  2(yl-y2){xlul  -x2u2)  (8) 

Consider  the  following  cooperative  control  Lyapunov  function  (CCLF)  candidate: 

V  =  \s2x+\s22+^s2r  (9) 


where  driving  this  value  to  zero  will  establish  the  following  three  invariant  manifolds  which  in  turn 
specify  desired  first  order  stable  error  dynamics  regarding  e} ,  e2 ,  and  er  with  a  relative  weighting  of  kr : 
sl=el+  \ex ,  s2=e2+  X2e2 ,  sr=er+  \rer .  In  order  to  drive  the  CCLF  to  zero,  take  the  derivative  with 
respect  to  time  and  then  force  it  to  be  negative  definite.  Following  the  notation  of  Primbs  et  al  [81,82,26]: 

v  =  K'X  +  =  ^f(x,)  +  0(x,)U|  +  Vlf(x2)  +  v;b(x2)u2  =  Slsl+sA  +krsrsr  (10) 


A  substitution  of  appropriate  relations  given  by  the  equations  above  results  in: 

V  =  fx  +bxiix  +f2  +b2u2  +  2fr  +brlux  +br2u2  (11) 

where  fx  and  bx  are  functions  solely  of  states  regarding  vehicle  1 ,  and  f2  and  b2  are  functions  solely  of 
states  regarding  vehicle  2: 

fx  =  Si [2r,2  +  2 'klrf  -  2 rf  (x,x,  +  y,y,  f  +  2(xf  +  yf  )]  (12) 

k  =2st(xtyl  -x,v,)  (13) 

f2  =  si  L2r2  +2  X2r2r2-  2  r~2  (x2x2  +y2y2)2  +2(x2  +y2)\  (1 4) 

b2  =2 s2(x2y2  -x2y2)  (15) 


Functions  fr ,  brX  and  br2  contain  the  states  of  both  vehicles  and  represent  the  coupling  introduced 
between  the  vehicles  by  virtue  of  the  desired  error  dynamics  prescribed  by  invariant  manifolds  sx ,  s2  and 
sr .  These  coupling  functions  are  given  by  the  following: 
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(16) 

(17) 

(18) 


2 fr  =krsr[2(xl  -x2)2  +  2(j,  - y2)2  +  MJ 
K  =  KSX 2(xi  -A'2)(-j|)  +  2(>>  -  v2)(i,)] 

6r2  =  [2(x,  -x2)(v2)  +  2(j,  -  72  )(-x2 )] 

Note  that  fx  ^  ,  etc.  The / and  b  notation  has  been  adopted  due  to  the  fact  that  these  functions 

express  the  dynamics  of  the  overall  cooperative  system  in  a  state  space  representation  of  the  desired 
cooperative  task.  We  now  wish  to  ensure  that  V  as  given  by  Equation  (1 1)  is  negative  definite  by 
dividing  this  task  up  in  some  kind  of  symmetric  way  between  the  two  vehicles  and  in  a  way  so  that  a 
vehicle  can  utilize  information  about  the  other  vehicle  (in  the  form  of  states,  a  combination  of  states,  or  a 
subset  of  states)  in  the  same  manner  that  the  other  vehicle  (identical  and  doing  the  identical  task)  uses 
information  about  it.  Likewise  we  could  consider  the  asymmetric  case  where  one  vehicle  has  more 
responsibility  for  the  task  than  the  other,  but  this  is  not  currently  under  consideration  (although  such 
considerations  would  be  appropriate  for  heterogeneous  teams  of  vehicles).  We  will  assume  that  each 
vehicle  has  full  state  feedback  of  its  own  states.  One  sufficient  condition  for  accomplishing  this  is  to 
separate  V  into  two  symmetric  contributions  and  then  make  each  contribution  negative  definite.  The  sum 
of  two  negative  definite  functions  is  in  turn  a  negative  definite  function.  Consider  the  following 
separation  of  V  into  two  symmetric  contributions: 

V  =  /l  +  fr  +  (fcl  +  K\)U\  +  fl  +  fr  +(^2  +  ^2^2  =  ^1+^2  O^) 

V ehicle  1  responsible  for  enforcing  Y ehicle  2  responsible  for  enforcing 

this  portion  negative  definite  this  portion  negative  definite 

f,  =fl+fr+(bl+bri)ul  (19b) 

V2=f2+fr+(b2+br2)u2  (19c) 

The  division  of  V  into  these  contributions  constitutes  the  formation  of  the  time  derivative  of  the  ACF’s. 

In  order  for  each  vehicle  to  make  it’s  contributed  ACF  rate  negative  definite,  each  needs  information 
regarding  the  other  vehicle  as  dictated  by  functions  fr ,  brl  (for  vehicle  1).  Discussing  this  information 
sharing  from  the  point  of  view  of  vehicle  1  ’s  responsibilities  toward  Equation  (19a),  it  will  need  the 
following  information  about  vehicle  2:  02 ,  x2 ,  x2,  y2  and  y2 .  An  important  point  general  to  cooperative 
control  among  vehicles  with  knowledge  about  the  other  vehicles  is  the  following:  vehicle  1  has  the  ability 
to  use  it’s  knowledge  of  vehicle  2’s  dynamics  in  order  to  obtain  x2  and  y2  from  02 ,  x2  and  y2  by 
utilizing  vehicle  2’s  known  dynamic  equations,  Equations  (la).  Therefore  vehicle  1  can  exploit  the  fact 
that  it  knows  the  dynamic  behavior  of  vehicle  2  in  order  to  construct  an  observer  (in  general  this  may  also 
require  knowledge  of  vehicle  2’s  control  u2 ).  Once  the  required  information  is  obtained,  each  vehicle  can 

enforce  the  following  relationships  in  order  to  make  their  respective  contributions  to  V  negative  definite: 

Choose  ux  to  enforce  \fx+fr  +  (bx  +  brX)ux  =  -Kx \bx  +  brl\^\sx  +^-s2r  (20a) 

Choose  u2  to  enforce :  f2  +  fr  +  ( b2  +  br2)u2  =  -K2\b2  +  br2\^\s22  +^s2r  (20b) 

where  the  right  hand  side  of  Equations  (20)  are  negative  definite  by  choice  of  positive  coefficients  Kx  and 
K2 ,  selected  to  account  for  modeling  error  bounds  in  the  typical  sliding  mode  control  sense.  Also  note 

(from  the  V  term  of  Equation  (19))  that  each  vehicle  only  acts  based  on  its  contribution  to  the  positive 
definite  scalar  metric  of  the  overall  global  error  given  by  V  (this  must  be  the  case  given  that  ux  cannot 

influence  \s\  and  vice-versa).  A  typical  run  of  this  control  law  formulation  is  shown  in  Figure  5  (values 
used  were:  v  =  1 ,  Xx  =X2=  2n/200  ,  Xr  =  2n / 50 ,  hr  =  3  ,  Kx=  K2=  0.3  ,  rd  =  5  ,  with  ux,u2  e  [-1, 1]  ). 
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Figure  5:  Control  performance  of  the  PF/CCLF/ACF  approach  in  cooperatively  establishing  an 
orbit  of  two  vehicles  around  a  target  (0,0)  with  a  specified  radial  distance  and  a  specified  angular 
separation  (relative  distance).  The  plot  on  the  left  shows  the  trajectory  of  the  two  vehicles  in 
establishing  the  cooperative  orbit,  and  the  plot  on  the  right  shows  a  snapshot  of  the  desired  relative 
angular  separation  of  90  degrees  after  the  vehicles  have  converged  to  the  desired  cooperative  orbit. 


Several  comments  and  discussion  points  can  be  made  regarding  the  toy  problem  above.  With  regard  to  the 
hypothesis  of  unifying  the  idea  of  a  potential  field  with  the  idea  of  a  control  Lyapunov  function,  the  above 
example  illustrates  that  the  “potential”  in  this  case  is  the  CCLF  of  Equation  (1 1).  The  manner  in  which 
this  potential  is  “processed”  by  each  vehicle  is  the  manner  in  which  each  enforces  Equations  (20).  This  is 
similar  to  the  physical  case  of  say  gravitational  potentials  specifying  the  force  as  the  gradient  of  the 
potential.  Although  a  gradient  with  respect  to  space  is  not  explicitly  employed  here,  the  thinking  is  similar 
in  that  a  scalar  field  is  utilized  to  derive  the  control  influence.  There  is  however  an  important  distinction 
between  physics-based  potentials  and  the  “potential”  used  here;  the  global  “potential”  here  as  represented 
by  Equation  (9)  is  divided  among  the  vehicles  and  in  effect  creates  a  separate  potential  for  each  to  follow 
that  is  unique  to  an  individual  vehicle.  This  could  be  seen  as  an  extension  to  the  idea  of  deriving  the 
“force”  from  the  gradient  of  a  potential  field  to  something  more  general  as  represented  by  Equations  (19). 
Viewed  from  a  controls  standpoint,  the  control  law  seeks  to  place  the  system  on  the  invariant  manifolds 
sx ,  s2  and  sr .  Once  on  the  manifold,  the  error  converges  to  zero  as  specified  by  the  desired  error 
dynamics.  An  important  piece  of  the  problem  is  however  missing  in  this  toy  problem;  the  invariant 
manifolds  sx ,  s2  and  sr  may  not  represent  achievable  trajectories  in  the  error  space.  This  is  due  to  two 
reasons  for  this  specific  case.  First,  the  non-holonomic  dynamics  are  not  accounted  for  in  specifying 
achievable  error  dynamics.  This  problem  has  been  addressed  extensively  by  Kyriakopoulos  et.  al.  [si],  and 
others  in  the  literature.  The  second  inadequacy/inaccuracy  of  the  above  example  is  that  the  control 
constraints  have  not  been  explicitly  accounted  for  (turning  rate  limits).  Papers  by  Wu  and  Jayasuriya  [io4] 
and  others  have  addressed  this  concern.  In  particular,  Astrom  and  Furuta  [2]  have  shown  that  an  inverted 
pendulum  can  by  swung  up  under  hard  input  saturation  constraints  by  controlling  the  energy  of  the 
pendulum.  This  hints  at  the  fact  that  given  the  right  metric,  or  PF  in  our  case,  even  input  constraints  can 
be  explicitly  accommodated.  Ogren  and  Leonard  [72]  address  similar  concerns  in  their  formulation  of  a 
navigation  function.  Further  evidence  that  the  example  problem  has  not  been  cast  in  a  strictly  proper 
sense  can  be  observed  from  Figure  5b  by  the  fact  that  although  the  system  came  close  to  the  desired 
specifications,  exact  tracking  was  not  achieved  (desired  radius  and  phase  both  contain  steady  state  error). 


The  most  important  point  to  take  away  from  the  simple  example  problem  presented  above  is  that  a 
method  is  proposed  for  the  cooperative  control  of  multiple  dynamic  systems  that  explicitly  incorporates 
interaction  among  vehicles.  The  idea  behind  the  proposed  method  elucidates  the  notion  of  defining  an 
“agent”  as  a  symmetric  contribution  to  decreasing  the  CCLF  in  a  manner  that  transforms  the  problem 
from  a  traditional  centralized  multi-input,  multi-output  control  formulation  into  one  which  distributes  the 


11 


control  task  among  physical  units.  The  outline  of  the  method  followed  in  the  example  problem  also  sheds 
light  on  to  the  information  required  to  be  shared  among  vehicles.  The  preliminary  method  additionally 
sheds  light  on  the  notion  of  defining  potential  fields  in  a  more  general  sense  than  those  inspired  by  purely 
physics-based  potential  fields.  The  problem  above,  for  example,  can  implicitly  incorporate  collision 
avoidance  between  the  vehicles  by  enforcing  a  relative  separation  distance.  As  presented  above,  this  was 
not  strictly  the  case  due  to  the  way  the  CCLF  was  split  between  vehicles;  each  vehicle  was  responsible  for 
decreasing  the  weighted  penalty  of  being  far  away  from  the  desired  radius  and  of  being  at  a  distance  other 
than  the  desired  relative  separation  distance.  Due  to  this  weighted  penalty,  it  was  not  enforced  that  each 
vehicle  monotonically  decrease  its  relative  separation  distance  error  -  but  it  could  have  been.  Such 
concerns  shed  further  light  on  how  to  split  and  enforce  the  contributions  of  the  CCLF  among  agents. 
Lastly,  the  proposed  approach,  if  cast  correctly  to  take  into  account  the  deficiencies  noted  above,  is  based 
in  stability  theory  and  can  therefore  offer  rigorous  analysis  tools  concerning  performance  and  robustness 
of  both  the  system  as  a  whole,  and  the  vehicles  or  agents  individually. 

However,  these  points  being  stated,  many  unanswered  and  open  questions  remain.  Exactly  how  can  the 
CCLF  and  PF  be  defined  to  explicitly  incorporate  or  address  dynamic,  state  and  input  constraints?  Are 
their  better  candidate  CCLF’s  that  would  reduce  the  required  amount  of  shared  information,  and  how 
would  such  CCLF’s  be  constructed  in  general  to  achieve  this  (since  ideally  each  vehicle  would  only  need 
to  know  the  current  scalar  value  of  the  CCLF  and  its  own  states)?  How  can  this  preliminary  approach  be 
cast  to  design  some  sense  of  optimality  by  linking  the  CCLF,  which  through  the  observations  of  Primbs 
[8i,82]  is  a  scaling  of  some  value  function  corresponding  to  some  unknown  cost  function,  to  properties  of  its 
cost  function.  What  happens  when  this  CCLF/PF/ACF  method  is  applied  to  a  general  number  of  vehicles? 
What  happens  in  terms  of  robustness  when  team  vehicles  are  added  or  subtracted  as  the  task  is 
progressing,  and  how  would  this  be  accounted  for  in  the  structure  of  the  approach?  If  a  state  observer  is 
employed  by  a  vehicle  to  estimate  the  states  of  another  vehicle,  how  does  this  need  to  be  incorporated  into 
the  overall  structure  to  ensure  stability  and  robustness?  These  and  other  questions  will  be  answered. 

5.  Additional  Documentation 

Several  simulation  files  have  been  generated  by  this  work.  Below  is  a  list  of  the  most  relevant  and 
finalized  simulations. 

coop_orbit  6  .  mdl  :  a  stand_alone  simulation  that  shows  the  robustness  of  assigning  two 
vehicles  to  a  CGMTE  sensor  fusion  scenario.  A  manual  switch  is  located  with  the  “target” 
block  to  change  from  a  smooth  target  track  to  one  with  sudden  90  degree  turns. 

coop_orbit  6_plot .  m  :  shows  an  animation  and  is  meant  to  be  run  directly  after 
coop_orbit6.mdl  is  executed. 

load  CGMTE_run_xx .  mat:  nine  data  files  of  the  assignment  of  5  UAV’s  to  3  targets  in  a 
CGMTE  scenario.  Generated  by  CGMTEassignment  which  calls  a  number  of  files. 

CGMTE_plot .  m  :  shows  an  animation  of  the  data  in  CGMTErunxx.mat. 
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